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Abstract 

In an ideal distributed computing infrastructure, users would be able to use diverse distributed 
computing resources in a simple coherent way, with guaranteed security and efficient use of shared 
resources in accordance with the wishes of the owners of the resources. Our strategy for approaching 
this ideal is to first find the simplest structure within which these goals can plausibly bo achieved. This 
structure, we find, is given by a particular recursive distributive lattice freely constructed from a presumed 
partially ordered set of all data in the infrastructure. Minor syntactic adjustments to the resulting 
algebra yields a simple language resembling a UNIX shell, a concept of execution and an interprocess 
protocol. Persons, organizations and servers within the system express their interests explicitly via a 
hierarchical currency. The currency provides a common framework for treating authentication, access 
control and resource sharing as economic problems while also introducing a new dimension for improving 
the infrastructure over time by designing system components which compete with each other to earn the 
currency. We explain these results, discuss experience with an implementation called egg and point out 
areas where more research is needed. 



1 Introduction 

An ideal distributed computing infrastructure would present diverse distributed computing resources in a 
conceptually coherent framework while guaranteeing flexible, secure, efficient shared use of the resources, all 
in accordance with the wishes of the owners of the resources. Although these goals have an obvious attraction 
and importance, the all-encompassing nature of the problem makes it difficult to approach as a whole. Our 
strategy here is to attempt to identify minimal required properties of the infrastructure until a simplest 
solution is essentially uniquely determined. We then hope that this solution is also sufficient for distributed 
computing in general. Starting with the simplest possibility, we consider an infrastructure made from objects 
of a single type called "caches." Taking storage of caches as a minimal requirement, we find that caches 
must be a particular recursive distributive lattice freely constructed from an assumed partially ordered set of 
all data which might be stored in the system. Minor syntactic re-arrangement of the resulting cache algebra 
yields a language and a single user environment which resembles a UNIX shell where "shell commands" pipe 
cache streams rather than byte streams. The algebra comes with a built-in concept of execution and a built 
in universal interprocess protocol which is useful on a large scale where the infrastructure is inevitably made 
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of independent communicating processes. Within the cache-based infrastructure, authentication, resource 
allocation and access control are treated as economic transactions where an explicit hierarchical currency is 
exchanged. Although an economic solution to these problems is not dictated as the unique simplest solution 
in the same sense that caches are, we argue below that a substantial simplification is obtained compared with 
conventional alternatives. At the same time, the currency opens a new dimension for system improvement 
over time: caches can be designed to compete with each other to earn the currencies so that cache "self 
interest" results in improved system performance. 

In order to test whether caches, the language and the currency really are sufficient for distributed com- 
puting in general, we have implemented these structures in a system called egg^ . Experience with egg shows 
(section 6) that our hopes are justified in the sense that this framework appears to be suitable for everything 
that one normally wants a distributed computing infrastructure to do. The status of egg as the "simplest 
distributed computing infrastructure" in the sense explained below gives some indication that cache-based 
infrastructure will be more flexible, easier to comprehend by users and easier to make secure than alternative 
systems. In the sections which follow, we define the infrastructure, discuss experience with egg and point 
out areas where more work is needed. 



2 Caches and data 

Taking the simplest possibility first, consider an infrastructure where all functional elements are objects of 
a single type and let us agree to call these objects caches. Since any distributed computing infrastructure 
must at least include storage, storage of caches in other caches must be possible. This means that at least 
two cache- valued binary operators on caches must exist: one to store a cache in another cache and a second 
to retrieve a stored cache by providing a cache as specification. These operations are related to each other 
by formulas which follow from what one intuitively means by "storage." For example, one expects 

retrieve(yl, store(A, B)) ^ A (1) 

since, if you store cache A in B, retrieving A should again produce A. Similarly, we expect that storing 
A in B followed by storing C in the result should be the same as storing C in _B followed by storing A in 
the result and so we expect that store is associative. Continuing this way, one finds that our expectations 
are exactly met provided that caches are a distributive lattice^] with store and retrieve as join and meet 
respectively. At this point, we know that caches must form a distributive lattice, but which one? A clue 
comes from considering the partially ordered set D of all data that one might want to ever be stored in the 
infrastructure. Since elements of D are to be stored, we want a distributive lattice to be constructed from 
D in a, way which "preserves D and its partial ordering." This informally describes what is known as a "free 
construction" in category theory. Limiting ourselves to such free constructions is usefully constraining since 
free constructions are unique in any category [5]. We are thus lead to search for the free distributive lattice 
constructed from a given partially ordered set P. This is given by the set of antichainff subsets of P with 
join and meet defined by 

AV B = {A(JBy (2) 

A A B ^ {A'^ n B-^y (3) 

where A^ {a,n antichain) is the set of elements which are maximal in A[S] and A^ is the set of elements in P 
which are less than or equal to some element of A. 

Proposition 2.1. Antichain subsets of P with joint and meet as defined is a distributive lattice. 
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Proof. First, let A, B and C be subsets of P and note that (A^)^ = , {A^Y = A\ {Ail B)^ = {A'^UB^)\ 
(AUB)^ = (A^UB^) and (AnB)^ = (A^nB^). Meet and join arc obviously commutative and idempotent. 
They are also associative, since, for antichains A,B,C C P, AW {BW C) ^ {AU BUC)^ and AA{B AC) = 
{A^nB^ n CJ-)T. Absorption laws follow since A\/{AAB) ^ {AU {A^ (1 B^)^)^ = {{A^)^ U (A^ n J5J-)T)T = 
A^T ^ = Aand, similarly, AA^AvB) = A. Lastly, AA(BVC) = (AJ'n((SUC)T)J-)T = (Ain(BUC)^)T = 
{A A B) V {A A C) and we have a distributive lattice. □ 

In addition to being a distributive lattice, the lattice of proposition 2.1 is also the free distributive lattice 
from the category of partially ordered sets and order preserving functions to the category of distributive 
lattices with V-preserving morphisms [4i{x V y) = (j){x) V (f>{y)]- To see this, let .F(P) be the lattice of the 
proposition, let a : P — *■ J^{P) map p to {p\ and suppose that / : P — > Af is an order preserving map to 

some distributive lattice M. Then F{A) = \/ aeAfio) is the unique V-morphism such that F o a ~ f. As is 
true in all categories, T{P) is the unique distributive lattice having these properties. 

The most straight-forward use of the free construction would be to let caches be antichain subsets of 
D itself. Although this does produce a distributive lattice, the resulting caches cannot easily represent, for 
instance, a hierarchical file system. The simplest plausible solution is then to define caches C recursively as, 
roughly speaking, the free distributive lattice on the partially ordered set D x C with (d, c) < {d' , c') if and 

only if d < rf' and c < c'. More carefully, let Co = {{}} and, for n > 1, let C„ = JF(Z3 x C„_i) so that C„ 
axe caches with depth less than or equal to n. Noting that Co C Ci C . . . , we can define caches in general 
as follows. 

Definition 2.1. Caches are the distributive lattice IJ^q^"- 

It is sometimes convenient to use algebraic notation for caches, writing, for example. 



for the antichain {(a, X), (5, Y), (c, Z)} and writing "0" for the minimum empty antichain. Generally speak- 
ing, the fewer the minimal elements in D, the more caches tend to collapse. For example, if D is the 
integers with their usual ordering (so that no integer is minimal), then (1,0), (2,0) and (1,(2,0)) V (2,0) 



are caches and (1, 0) A [((1, (2, 0)) V (2, 0)] = [(1, 0) A (1, (2, 0))] V [(1, 0) A (2, 0)] = (1, 0) V (1, 0) = (1, 0). 



On the other hand, if D are strings with trivial ordering (so that all strings are minimal) , then caches like 
{hello, 0) V {world, 0) cannot be reduced. Non-minimal elements of D are useful for selection, for example if 
D is the set of strings with world < wor*, then [{hello, 0) V {world, 0)] A {wor*, 0) = {world, 0). With these 
examples in mind, the data D used for the infrastructure should have many minimal elements to provide 
a wide variety of stable caches and a rich order structure to provide flexible selection and database-like 
functionality. As we shall see, it is advantageous for D to allow lazy refinement and to allow new kinds of 
data to be dynamically included in D as they are encountered. Of course, easy user specification of elements 
of D is also an important criterion. 

With simple user specification in mind, construct D from a meet semilattice direct sum|B] 



of meet semilattice basic data types do, . . . , d„ with minimum elements Oq, Oi, . . . , 0„ respectively. A user 
can thus specify an clement of A by providing zero or more type/value pairs, such as text: hello, size: 5, 
{text: hello, size: 5} or http:www.bu.edu, where text, size and http are meet semilattices. Since any 
partial order can be made into a meet semilattice by adding a minimum element if necessary, the basic data 
types only need to be sets with some conveniently chosen partial order. 



{a,X)V{b,Y)V{c,Z) 



(4) 



A = doQ)di®---®d, 



"n 



(5) 
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Before completing the construction of D from A, we need some preliminary results concerning "exten- 
sions" of meet semilattices which will provide lazy refinement of elements of A. 

Definition 2.2. An extension of meet-semilattice S is a mapping F{s) = s A /(s) where f : S ^ S is an 
order preserving function. 

It is easy to see that extensions are order preserving, non-increasing and closed under composition. Given 

a set E = {Fq, Fi. . . . , _F„} of extensions of meet semilattice A. G A is a fixed point of E if F{d,) = d 
for all F in E. Such fixed points are unique, for suppose that F{d) and G{d) are fixed points of d. Then 
G{F{d)) = F{d) = d A f{d) = dA fid) A g(d A fid)) < dA fid) A gid) A gifid)) < dA gid). Therefore 
Fid) < Gid). Similarly G(rf) < Fid). 

With the above results, suppose that we have a set £ of A extensions and a guarantee that every d £ A 
reaches a fixed point of E after a finite number of extensions. Since fixed points are unique, we can let 
elements of A be equivalent if they have the same fixed point and let D be the equivalence classes with 



so that D is clearly a meet semilattice provided that A is a function. This is guaranteed by proposition 2.2. 
Proposition 2.2. D is a meet semilattice. 

Proof. First note that for a.b e A, if i^(a A b) is the fixed point of a A 6, then F(a A fe) = FiFia) A Fib)). 
This follows because F(a A 6) < F(a) A implies F(a Ab) < FiFia) A Fib)) and F(a) A Fib) < aAb 
implies FiFia) A Fib)) < i^(a A b). Next, note that for any two points a and b in A, there is a composition 
of extensions which takes both a and b to their corresponding fixed point. To see this, let F take a to 
its fixed point and let G take F(5) to its fixed point. Then G o F takes both a and b to their fixed 
points. Finally, let a ^ a' , b ^ b' and let F take a, b and a A 6 to their respective fixed points. Then 
F(a' A b') = FiFia') A Fib')) = F(i^(a) A F(6)) = i^(a A b) and the meet defined above makes D a meet 
semilattice. □ 

Finally, demote D from a meet semilattice to a partially ordered set by removing any element containing 

a minimum element of the basic data types do, rfi, . . . , (i„. The resulting partially ordered set has a single 
maximum {} and, as desired, has many minimal elements consisting of direct sums of covers of Oq, Oi, . . . , 0„, 
one from each of do, di, • • • , dn- 

This choice of D has the features needed for the infrastructure including easy user specification, many 
minima and rich ordering structure for selection purposes. At the system level, an element of D can be 
represented as the fixed point of an element of d G A. Since fixed points are preserved by the extensions, 
extensions can lazily be applied to d whenever needed and new extensions can be dynamically added to the 
system without requiring recomputation of existing data. 

3 Cache algebra 

Given any set do,di, . . . ,dn of "basic data types" in the form of meet semilattices, we have defined a 
concrete partially ordered set D and a corresponding uniquely determined distributive lattice of caches given 
by definition 2.1. The pair structure of caches suggests defining two additional operators /,// :CxZ)— >C 
which "select subcaches" of a cache given d G D. Let 



[a] A [b] = [aAb] 



(6) 



(^0 V V . . . )/d = liAo/d) V iA^/d) V . . .] 



(7) 



4 



{a,X)/d = X A{d,l), 
(Ao V V . . . = [{Ao//d) V {Ai//d) V . . . ] 



(8) 



(9) 



and 



{a,X)//d^[{a,X)A{d,l)]V{X//d). 



(10) 



Given a cache {a,X), for example, {a,X)/{} — X is the "contents" of {a,X). If D are strings with 
world < wor* as before and C = {a, {hello, 0) V {world, 0)), then C/wor* = C//wor* = {world, 0) and 
C//{} = (a, (/te//o. 0) V {world, 0)) V {hello, 0) V (woWd. 0). 

The action of the operators V, A, / and // is completely prescribed by the definitions given here and 
can be implemented once for all caches. The cache algebra is completed by adding one more binary operator 
<: C X C — > C (called "put," not to be confused with "less than") which, on the other hand, has more 
flexibility. Put is required to distribute over V from the right as in (A V B V C) < Y = {A < Y) W {B < 
y) V (C < Y) so that its action is determined by its action on singleton caches which is defined by 



where put is only required to be a cache valued function of its arguments. It is convenient to organize the 
action of put by introducing a data type type which specifies a "singleton cache subtype" in the sense that 
singleton caches with different type values have different actions under put. A storage cache, for instance, 
might have put(typc : storage, X, Y) = X\/Y while a counting cache might have put(typc : counter, X, Y) = 
(count : #y, 0) counting the number of singleton caches in Y. The egg infrastructure has hundreds of such 
singleton cache subtypes performing many different tasks. Since put is not generally associative, we take 

a < b < c = {a < {b < c)) grouping from the right in the absence of parenthesis. 

The natural language for using caches is just cache valued expressions made with operations V, A, /, // 
and <. Here, for example, is a typical cache- valued expression 



where X and Y are caches and depth and name are data types. In order to keep the user experience as 
simple as possible, we would like to simplify the language syntactically and, if possible, hide the underlying 
lattice concepts in a more intuitive and familiar framework. In view of this, introduce three reference caches: 
"." (current working cache), "~" (home cache) and "0" (home caches of people known to you) and introduce 
a set of caches with conventional short names like pwd, cd and Is. Proceeding informally, replace 

• X/{} with X/; 

• {name:foo} with foo; 

• (X < Y) /{} with X Y if X is a named cache; 

• (d,0) with d and 

• V with ' ' 

to define a language called egg shell (the exact language definition can be found in Appendix A). Using these 
replacements, and letting X = Is and Y = ., equation 12 

Is depth: 3 ./*.py 



{d,X) <Y = {d,put{d,X,Y)) 



(11) 



{X < ((depth : 3,0) V y/{name : *.py}))/{} 



(12) 
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has a syntactically reassuring resemblance to a UNIX shell command. To see that the resemblance is not 
superficial, notice that the cache algebra also has built in concepts of "execution" and "piping between shell 
commands." Any cache A may be "executed" by computing A/, in other words, by computing its contents. 
For example, entering Is . into the egg interpreter causes it to compute 

(I < (~/shells/ls < .)/)/ 

so that the contents of (~/shells/ls < .)/ arc put into the interpeter where they are explicitly written 
to terminal output. When multiple shell commands are used, they effectively "pipe their output into each 
other from right to left." For example, a typical egg interpreter command 

. < music < has ext:mp4 ex depth:3 web. google "free music" 

uses egg shell commands web. google (which comes from the web plug-in), ex and has (which come from 
the yolk plug-in) to cause the contents of a Google search for "free music" (web . google) to be expanded 
to depth 3 (ex depth: 3) and puts any mp4 files (has ext:mp4) into a new cache named "music" which is 
then stored in the current working cache ".". For a more database-like example, 

count has tstart: class lines ~/egg/plugins/*//ext :py 

uses the lines shell command to split any text data from the egg source code (~/egg/plugins/*//ext :py) 
into individual lines, selecting only those lines which begin with "class" (has tstart: class) by extending 
text data (which has trivial ordering) to tstart data (which has prefix ordering). As a result, this command 
counts the number of classes in the egg source code. 

Searching for the simplest distributed computing infrastructure, we find that the assumption that only 
one kind of object is involved combines with the need to do storage to imply that caches must at least form 
a distributive lattice. Arguing that caches should be freely constructed from the partially ordered set of 
all data, we come to an explicit free distributive lattice once the partially ordered set of all data is chosen. 
Adding operators / and / / for convenience and < to allow caches to do more than storage, we have the 
"essentially unique" simplest candidate for a general purpose infrastructure. Through experience with the 
egg implementation, we find that caches as described here are able to conveniently represent all the functional 
elements of what one wants in a distributed computing infrastructure including files, directories, web sites, 
data bases, batch jobs, running processes, servers and many others. 

From the perspective of a single user, the infrastructure resembles a database-integrated distributed 
operating system where UNIX-like shell commands pipe cache streams rather than byte streams and where 
the global set of all caches takes the place of the file system. The normally difficult problem of cache 
coherence in distributed file systemsjZl |8j |9] is relegated to implementation details of the few cache subtypes 
which need it, such as banks. Other than caches with such special implementations, changes made to caches 
in one egg session are not reflected in other egg sessions without an explicit save command. Each egg session 
is "editing the internet" so that two users may modify the same cache, may see different results and only the 
last person to save is guaranteed to see the same results in a new session. Since caches are self-referential 
but not hierarchical, flle system content smoothly combines with web content, bridging the gap between the 
Plan-9 like "everything is a file" viewp^ and attempts to abstract server-based functionality such as web 
services [TT] and the semantic web[T2]. Compared with UNIX, more combinations of shell commands pipe 
into each other usefully and one seems to need fewer shell commands and fewer optional "fiags." Typically 
zero or one flags are needed per egg shell command. 
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4 Servers, cache addressing and home caches 



On a large scale, the infrastructure described here is a collection of independent processes representing 
personal interactive sessions or servers. Within a particular process, the other processes in the system are 
represented by caches. The most naive behavior of such caches would be to execute whatever cache is put 
into them by the client. At the interprocess communication level, each server would 

1. Receive a serialized cache X; 

2. Return the serialization of X/. 

Since cache execution is sufficient to do any operation, this would give the client complete control of the 
server. Real servers use a simple variation of this where the returned results go through a server egg shell 
command on the server side: 

1. Receive a serialized cache X; 

2. Return the serialization of server X; 

so that a server process appears to the client as a cache which yields server X when X is put into it. 
This formulation of server behavior leaves room for improvements to clients and servers over time while 
keeping the byte stream level protocol unchanged. Although servers are defined here as executers, they are 
typically not directly used that way. Instead, base class implementations of the cache algebra use server 
cache execution to implement V, A, /, //, and < so that caches hosted by remove servers appear seamlessly. 
Given a particular singleton cache {d,X), its operations are implemented by a server if d contains a server's 
personal public key, IP address and port number. Existing protocols like SSH and HTTP are included as 
artificial persons so as to fit in the same framework as the servers described here. 

Prom the egg shell perspective, standard protocols and egg server caches appear uniformly and can be 
referred to using basic internet data such as 

cd http:www.bu.edu 

or 

Is {person: Alice, port : 33366 , host : physics .bu. edu, path: home}/ 

where one refers to host names or IP addresses explicitly. Alternatively, it is often more convenient to ignore 
low level addressing and to instead refer to caches relative to the "home caches" of persons, organizations or 
servers known to you. A person, server or organization creating such a public/private key identity may also 
provide any egg shell expression defining their home cache. Home caches may be simple web pages such as 
http : www . bu . edu or may be more sophisticated such as 

first hasnt d:error test {http:www.bu.edu/~Alice} {http:physics.bu.edu/~Alice} 

which provides the first of two web pages which axe error-free. A home cache may also be the output of an 
egg shell command so that the home cache can be modified whenever the corresponding plug-in software is 
updated. The home caches of all persons and servers known to you appears in the "@" cache in egg shell, 

so one can navigate by 

cd Q/BostonUniversity 
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or 

cd O/Alice 

rather than by referring to exphcit hosts or IP addresses. In effect, home caches make a flexible common 
address space not shared globally, but shared by all who have a common public key in their rolodex. 

5 Economics 

A large scale infrastructure naively built along the lines described so far would have a number of problems 
in common with the present-day internet. For example, we have yet to propose a mechanism for sharing 
resources among multiple users in accordance with the wishes of the owners of the hardware. In conventional 
terms, we also, so far, lack any mechanism for authentication, access control, accounting or resource allocation 
and have made no mention of virtual organizations j22j . Instead of treating these issues individually, we 
proceed in two steps. First, introduce a common hierarchical currency to explicitly represent the intentions 
of actors within the system. Next, treat problems like authentication and resource allocation as economic 
problems to be solved by currency exchange. Within this framework, improved solutions to these problems 
can appear over time via improved cache designs without disrupting the infrastructure as a whole. At 
the same time, currency exchange opens a new dimension for system improvement by designing caches to 
compete with each other for currencies of interest. Over time, the design of such caches can be improved 
by using, for example, sophisticated auctions [T3] IHj and resource estimation techniques [TS] so that the 
infrastructure becomes increasingly internally efficient and increasingly responsive to the desires of people 
and organizations. 

Economics within the infrastructure is based on a hierarchical currency called the "egg." Eggs appearing 
in the form of a data type called check which can be minted by any user, transferred from user to user and 
which may be earned by caches. Each user has a bank cache which is used for storing checks and executing 
bank to bank transfers. Possession of a valid check 

100[Janl, 2011]BostoiiUniversity.Alice.Bob (13) 

for example, cryptographically guarantees that the person associated with the BostonUniversity private key 
minted (at least) 100 eggs, transferred these to Alice who then transferred 100 eggs to Bob. Transfers can 
be gifts, as above, or payments denoted by an underline as in 

100[Janl, 2011]BostonUniversitY. Alice. Bob. server 1 (14) 

where Bob has paid the entire 100 eggs to serverl. Depending on the intent of the owner of serverl, the 
server 1 bank might be configured to accept currencies according to 



currency 


who 


preference 


BostonUniversity. * . Alice . ? 


anyone given BostonUniversity currency by Alice 


1000.0 


BostonUniversity. * 


anyone possessing BostonUniversity currency 


50.0 


HarvardUni versify. * 


anyone possessing HarvardUniversity currency 


40.0 


* 


the general public 


1.0 



which effectively controls access by listing acceptable currencies and implies that serverl will prefer to earn 
BostonUniversity currency given by Alice over all others. Note that serverl can safely grant access to anyone 
who Alice decides to give currency to without having to know who these people are in advance and without 
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using a certificate authority. Payments made to serverl are stored in the serverl bank cache which then 
transfers the check back to the payer until the check returns to its original creator and becomes a complete 
receipt. 

To define the currency more explicitly, we presume a common cryptosystem with public keys, private 
keys and digital signatures. A payload is a string encoding a tuple consisting of a floating point currency 
denomination, a starting date, an expiration date, a tracking number and a payment bit. A "check" is a 
signed tuple of strings ending with a public key. The owner of a check is the owner of this public key. With 
these preliminaries. 

Definition 5.1. A check is either a payload/ public key pair (i, k) signed by anyone, or a payload/ check /public 
key triplet (L, c, k) signed by the owner of the check c. 

To illustrate the life cycle of a check, suppress the payload except for denomination and payment bit, let A, 
B and C be be identities (public/private key pairs) and denote text x signed by A by [x]^^. 



action 


check 


display 


A gives B 100 eggs 


[100,5]^ 


lOOA.B 


B pays C 50 eggs 


[50P,[100, B]a,C]b 


50A.B.C 


C returns check to B 


[50, [50P,[m,B]A,C]B,B]c 


50A.B.C.B 


B returns check to A 


[50, [50, [50P, [100, B]a, C]b,B]c, A]b 


50A.B.C.B.A 



Examining the second step in more detail, B begins with [100, B]a, constructs [50P, [100, B]a, C]b using the 
B private key and sends the resulting string to C. To preserve the initial 100 egg value, B is required to both 
produce and save [50, [100, i?]^, i3]_B, self-giving 50 eggs, and to destroy the original string [100, i?]^!. If the 
B bank follows this procedure, no new currency is generated or lost while making the payment. Each new 
payload created by the B bank keeps the existing start and expiration dates and is given a unique tracking 
number so that the check can be identified after it returns to the A bank. In the last table entry, the check 
returned to A has completed its circuit and is thus a receipt. 

Although the presumed integrity of digital signatures prevents checks from being forged, it is clear that 
some behavior that we need is only enforced by the software of the bank caches. In particular, handling of 
checks must satisfy banking rules: 

1. Other than minting a new check, all check operations must preserve total value; 

2. Checks with a payment bit set may not be used for additional payments; 

3. Cashed checks recieved by a bank must be transferred to the previous bank in the check sequence. 

Determined users could, however, violate the banking rules by modifying their own bank cache software 
or by copying checks outside of the system. Although such rule violations cannot be prevented, they can 
be detected after the fact by examining receipts and tracking numbers. It is not clear to us whether such 
problems should be discouraged, prevented by putting all banking caches on trusted servers or detected and 
punished. 

Unlike previous attempts to combine economics and distributed computingfl6 | [T71 [151 [TOl [20] . egg cur- 
rency is compatible with a wide variety of independent economic behaviors ranging from ignoring currency 
entirely to equally sharing among friends to traditional fixed monthly allocations to commercial situations 
where earning currency results in real financial gain. Unlike grid systems where virtual organizations are 
layered on top of distributed computing [22j, virtual organizations here appear organically: users, servers 
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and virtual organizations are the same thing. Take, for example, an organization like the ATLAS high en- 
ergy physics experiment. ATLAS needs a computing infrastructure which includes the 2500 physicists and 
169 universities and laboratories in the collaboration [21]. Within the infrastructure proposed here, ATLAS 
would be a single user like everyone else with a single private key, rolodex and bank. If, for example, AT- 
LAS decides to spend 90% of it's worldwide computing resources on Higgs searches and the remaining 10% 
on Supersymmetry for the next month, ATLAS would mint two checks 90 [Mar-1-2009] ATLAS . Higgs and 
10 [Mar-1-2009] ATLAS . Supersymmetry sent to the Higgs user and the Supersymmetry user respectively and 
continuing recursively until one reaches the people who directly spend the currency. Worldwide resources 
respond to the newly minted checks because ATLAS collaborators and computing centers want to earn the 
ATLAS currency because they want to contribute to the project. The ATLAS person can have this effect 
without having to know exactly who is in the Higgs group or who is using what resources when. At the 
largest scale, the currency makes a global certificate authority unnecessary (note that a user who gave any 
requested amount to any acceptable user is the same thing as a certificate authority). The egg currency, 
then, is intended to be used at all scales, from global applications to organizational applications to server 
authentication to setting access controls on a single file. Although our current egg implementation includes 
egg currency as defined above, only the most basic economic interactions have been used so far: user directed 
gifts and payments, two-way authentication by currency exchange and price-based access control. In spite 
of the fact that much theoretical progress has been made as to how caches can compete with each other in 
auctions |14j. basic questions remain as to how economically activated caches should be designed, how they 
should be collectively used and what properties such collections might have. Basic questions also remain in 
the macroeconomics of the egg currency: how should one optimally deal with bad behavior such as banking 
rule violations, inflation and caches which accept payment but do not provide expected service? It is impor- 
tant to design banks so that easy to understand monetary policies can be set and then executed over time 
with minimal direct user intervention. Since the economic system has an important social component, it is 
hard to make do without large scale testing involving a reasonable number of users. This has not yet been 
done. 



6 Implementation 

Starting with an attempt to make an infrastructure from only one kind of object, we have arrived at a 
concrete system consisting of caches, a cache algebra, a language and a protocol which is essentially uniquely 
determined in the sense described in section 2. Adding the egg currency provides a framework for authen- 
tication, access control, resource allocation and a proposal that system improvements over time come from 
designing caches to compete with each other to earn eggs. In order to understand whether this framework re- 
ally provides everything that one expects of a general purpose distributed computing infrastructure, we have 
made an implementation of the infrastructure called egg written in Python[23j. Caches, data types, D and 
singleton caches are implemented as Python classes which are organized into "plug-ins." Egg includes plug- 
ins for the core system: yolk and egg, for common protocols: ssh[24 , web [25 , gsi[22 , local[2B:, for batch 
schedulers: pbs^T,, Isf |,28j . for commmon unix software linux[29, . rpm[30., apache[31_, for mysql based 
servers mvsql[32]. lfc[33], and for the ATLAS high energy physics experimental], pacman|34j. panda [35]. 
dq2[36 , atlas[5r. When imported, each plug-in contributes new data types, extensions, singleton cache 
subtypes and egg shell commands to the system. Most of the plug-ins are small (typically a few hundred 
lines of Python) and are easy to write without detailed knowledge of the core system. Figure 1, for example, 
shows the complete Python code required to define an egg shell command lines which splits text into caches 
containing one line of text per cache. Including all 24 plug-ins, there are 194 egg shell commands, 439 cache 



10 



subtypes, 845 data types and 330 extensions in the system. Extensions as defined in section 2 play important 

import Pipe, Data 

from Pluglns import PI 

class Lines (Pipe. Pipe ) : 

"splits text into lines" 
command = 'lines' 

def iter (self) : 

for c in Pipe. Pipe. iter (self): 

if c . data ( ) . has ( ' text ' ) : 

for line in repr(c.data( ) .get( 'text' ) ) .split( ' \n' ) : 

yield self .inner(Data.Data(PI.datum( 'text' ) (line) ) ) 

Figure 1: Python source code for a functioning egg shell command called lines. Pipe is a singleton cache 
subclass where put{d, X,Y) is a function of Y only. Data is the Python version of the universal partially 
ordered set of data D. PI is the plug-in manager allowing the creation of new objects of selected basic data 
types. The command='lines' data member is picked by the plug-in manager causing a shell command 
named lines to automatically appear in the interpreter. For each cache c that extends to text data, the 
inner loop lazily produces one cache for each line of text using the self.inner function which creates caches 
from data. Most singleton cache subtypes such as Lines need only provide a subclassed Python iterator. 

roles in the system which we did not entirely anticipate. These not only efl^ectively save space and time via 
lazy evaluation, but they also simplify the introduction of new cache types and new egg shell commands. In 
using the infrastructure, we find that concious considerations of extensions or lazy evaluation isn't necessary 
and extensions work smoothly below the level of normal user awareness. The stat command is an example 
of unexpected benefits: stat is able to collect statistics for any data type which is an abelian group as well 
as a meet semilattice. It can, for example, compute statistics for file sizes by summing the size data which 
are put into it. Plain text data, on the other hand is not an abelian group and is ignored by stat. The 
following, for instance, 

stat d:size .// 

sums the file sizes of all files in .//, but, in addition, it sums all data types which extend from size, 
producing useful additional statistics about size (such as a histogram of sizes) without having to know that 
these statistics exist ahead of time. A typical use of this feature would be something like 

stat d:pbs pbs.jobs ssh: atlas. bu.edu 

which produces a summary of all PBS- related [27! data produced from PBS jobs running on atlas.bu.edu. 
We find that a substantial part of the system can be written just by writing extensions. 

One of the most useful features of the system comes from the simple observation that a text file containing 
egg shell code can either be executed as a script or can be interpreted as defining the contents of the text 
file itself so that one can, for example, "cd into a script." Files which are interpreted as contents rather 
than as a script to be executed are called "hatches." A typical hatch, for example, would be a file called 
hosts. hatch containing: 
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# 

# NET2 worker nodes 
# 

show d : linux . host , linux . load , linux . uptime 
linux. hosts (§/NET2/Boston/iiodes/ 

When a cache is created for the hosts .hatch file, it contains the output of the indicated linux. hosts egg 
shell command which produces one cache representing each of the Boston University worker nodes in the 
U.S. ATLAS Northeast Tier2 Center [571 . The line show d:name, linux. host, linux. load, linux. uptime 
defines default display options. The net effect of this is that if you cd into hosts, you see the linux hosts 
each with load and uptime information without having to remember that the linux plug-in contains the 
linux. hosts command or having to remember how to use it. Hatches make it easy to blend caches defined 
by standard protocols like HTTP and SSH with egg server caches with the output of egg shell commands 
and with other caches. This strongly re-enforces the impression of a uniform infrastructure spanning all 
content and functionality. 

The success of a uniform cache oriented view of all infrastructure content means that one quickly loses 
track of which protocols are used by which caches and which caches are hatches and which are not. This, of 
course, is the desired result, but it also means that any protocol- level errors which occur will be incompre- 
hensible to most users. In general, errors are treated in egg by producing error data rather than by raising 
exceptions in the interpreter. By default, high-level cache oriented error data are visible in the interpreter. 
This allows users to notice data of the error type, collect them and analyze them using the same tools which 
are used for analyzing any other kind of data. Creation of error data is typically accompanied by created 
errorbase (low level error) and traceback (Python traceback) data which can be also treated and analyzed 
like any other kind of data. We also found it convenient to introduce a logging cache, which is particularly 
useful for collecting statistics about running egg sessions. 

In addition to interprocess communication with servers, egg shell commands in this implementation are 
able to execute in parallel by forking and returning results from multiple child processes, allowing effective 
use of multi-core machines. An "execution network" keeps track of hosts which are reachable by protocols 
allowing Unix shell execution. By composing Unix shell commands, egg can effectively reach further than by 
SSH from a single account and can transparently use the entire network for extensions and can transparently 
use software like MySQL [32] and Globus [22| even if the software is not available where the egg interpreter 
is actually running. 

The egg infrastructure includes simple servers and a basic economic system which is capable of basic 
currency and banking operations, authentication and access control. Figure 2 shows a schematic of a simple 
egg server installed on Alice's PC. AlicePC accepts incoming messages either from someone who Alice 
has directly given currency to (providing Alice.?), or from someone holding any Alice, Bob or Boston 
University currency (providing Alice.*, Bob.* or BostonU.*). The server values these currencies with 
respect to each other in the ratios 1000.0/100.0/20.0/1.0 respectively as indicated in the figure with the idea 
that more advanced server designs would act to prefer earning higher valued currencies. As indicated. Bob 
communicates with AlicePC by sending a serialized cache X containing a payment 1 . OBob . AlicePC and 
a serialized cache to be executed on port 33366. AlicePC returns a receipt 1 . OBob . AlicePC . Bob to Bob 
(thus completing a two way authentication) and the serialization of X/ , the requested execution. The server 
operates by forking a process to handle each new connection. Alice's egg software installation is indicated on 
the left side of the figure. The figure indicates plug-ins being dynamically imported from the home caches 
of trusted persons egg, Alice, BUegg, HUegg and ATLASsoft. 
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AlicePC 




@/egg g /Alice @/BUegg @/HUegg @/ATLASsoft 



Figure 2: A schematic of a simple egg server as installed on Alice's PC. 



A number of applications of the system related to the U.S. ATLAS Northeast Tier 2 computing facilities 
at Boston University and Harvard University [37 . A typical simple ATLAS-specific operation is 

atlas .unsatisfied @/NET2/Harvard/nodes 

which uses the atlas .unsatisfied command from the atlas plug-in to test whether all of the Harvard worker 
nodes have the RPMs and libraries necessary for ATLAS Monte Carlo simulation jobs to run successfully. 
A typical more advanced command 

Stat d:unix tr din: text dout:path hasnt tstart:srm split sep: atlas. bu.edu \ 
tr din: If c .pf name dout:text @/NET2/DDM/lf c//since : yesterday 

uses five egg shell commands stat, tr, hasnt, split, tr to test if the path entries which have arrived since 
yesterday in an ATLAS-specific MySQL database and server called "LFC" really exist in the local file system. 
Reading from right to left, entries in the LFC database includes dates which extend to the data type since, 
which is a time interval partially ordered by inclusion. As a result, (§/NET2/DDM/lf c//since : yesterday 
contains all database entries which have arrived since yesterday (defines as the time interval from 24 hours 
ago to now in univeral time coordinates). The tr shell command extracts any Ifc.pfname data containing 
the URLs of files which are supposed to exist in our storage element. The shell commands split and hasnt 
are then used to manipulate the text value of the If c .pf namie so as to extract the UNIX path as text data 
being emitted by the hasnt command. The second tr command then translates the text path into path 
data so that it is interpreted aa the path of a file in the local file system. Finally, stat summarizes all of the 
UNIX level information about these files such as whether they exist, their size, creation/modification/access 
times, ownership and access control bits. 
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The egg system is developed enough to give a clear impression of the proposed infrastructure in a wide 
variety of applications. We find the situation encouraging in that all the needed elements of the infrastructure 
easily fit in the framework where everything is a cache and all added functionality is provided by cache put 
operations. Although the economic system has only been tested in the limited way explained in section 5, 
we have not found any needed feature which does not easily fit within the framework described here. 

7 Appendix: The egg shell language 

Egg shell is a very simple language with no variables, subprograms, fiow control, data structures, classes, 
assigments, functions or exceptions. It has only four binary operations: concatenation, /, // and <. Power 
and flexibility comes from an available set of egg shell commands. It is convenient to directly deflne the 
language as a recursive partial function tt mapping strings to caches. Strings which are not in the domain of tt 
are, by definition, not legal egg shell programs. The compiler tt is defined in terms of another partial function 
S mapping strings to data (elements of D) . Both tt and S are defined by sequences of partial functions where 
the action of the sequence on a string s is defined to be action of the first item in the sequence for which s 
is in the domain of the partial function. First, define S by the following: 



name 


pattern 


result 


left blank 


' 'x 


5{x) 


right blank 


x' ' 


5{x) 


comma 


x','y 


5{x)^5{y) 


curly 


{x} 


6{x) 


curly2 


x{y} 


{name : x} A 5{y) 


datum 


X : y 


x : y 


maximum 


II 


{} 



Given S, we can define the compiler tt. Compilation is always done within the context of a particular set of 
available egg shell commands. Let shell map egg shell command names to their corresponding caches. Let 
DOT, TILDE and AT be the caches corresponding to the three distinguished named caches "., and @" 
respectively and define tt by 



name 


pattern 


result 


lines 


X newline y, leftmost 


tt{x) V n{y) 


left blank 


' 'x 


'7t{x) 


right blank 


x' ' 


7r(x) 


shell command 


C x for command C 


(shell(C) < 7t{x))/{} 


put 


X < y, rightmost 


7r(x) < 7r(y) 


lub 


X y, leftmost 


7r(x) V 7r(y) 


parenthesis 


(x) 


7r(x) 


slashes 


x/d or x/ /d, rightmost 


Tr{x)/6{d) or TT{x)//6{d) respectively 


current working cache 




DOT 


context 




TILDE 


friends 


'@' 


AT 


singleton 


X 


{6{x),0) 



In text files, backslash line continuation and characters indicating comments are handled in the usual 
way. 
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